feat(mcp): report truthful index freshness from checkout evidence - #1561

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

Thank you @tmonestudio. The underlying issue is confirmed on current main: verbose index_status reports live checkout context, while the indexed generation stores no corresponding checkout identity. Persisting identity at the publication boundary is the right contract to review.

This is a broad correctness change despite the focused product claim: 11 files and about 1,300 added lines across git subprocess handling, MCP output, pipeline publication, store migration, and tests. It therefore needs a full storage/schema and fail-closed review rather than a quick UI pass. The current head is mergeable, but lint / lint and consequently ci-ok are red and the main test matrix was skipped. Please clear the lint failure before review.

Thank you for the reproduce-first coverage and legacy-database checks. The queue is full, so detailed feedback may take some time, but this is labeled and queued.

Checking in — this has been quiet since 12 August and it is still open and still wanted, so here is where it stands with the friction removed.

The lint blocker is six clang-format violations in two files:

src/store/store.c:3171:94
src/store/store.c:3172:34
src/mcp/mcp.c:4504:80
src/mcp/mcp.c:4505:57
src/mcp/mcp.c:4505:78
src/mcp/mcp.c:4506:60

make -f Makefile.cbm lint-format reproduces it, and running clang-format over just those two files fixes it. One caveat that has bitten people here: it must be the Homebrew LLVM build — a standalone clang-format-20 produces spurious whole-file drift.

And an honest note on the rest, so the delay does not read as all yours. Even with lint green, this is not a fast merge: 11 files and ~1,300 lines spanning git subprocess handling, MCP output, pipeline publication, store migration and tests. It needs a full storage/schema and fail-closed review, and that review is ours to do and has not happened yet. Clearing lint is what lets the main test matrix actually run — it was skipped entirely on the last CI pass, so nobody has seen this change tested.

The underlying problem you identified is confirmed on current main and the contract you proposed — persisting checkout identity at the publication boundary — is still the right shape. Your reproduce-first coverage and the legacy-database checks were noted at the time and still stand.

If you have moved on, say so and we will take it from here rather than leaving it to age. If not, clear the lint and the matrix will finally have something to say.

An update, and a decision that went in your favour.

#1727 independently added a top-level freshness to index_status as well — a string (fresh / stale / unknown) rather than your object, derived from clean Git snapshots taken before and after indexing. Neither of you could have seen the other: the PRs are independent, and git would have merged them without a conflict, leaving index_status with one key emitted twice at different types depending on merge order.

Your object stands. The reasoning is that a caller which learns why the graph is not current, and what to do about it, is better served than one that gets a single word — your reason, reasons[] and recommended_action are the difference. This PR is also standalone and already reviewed, so it does not wait on a six-PR stack. #1727 has been asked to drop its field and keep its last_index_attempt record, which is genuinely additive to yours.

One idea from that PR worth folding into yours if you touch the docs: "a dirty tree is never treated as evidence in either direction". Your implementation already behaves that way — a dirty tree cannot reach current because tracked changes force stale, and status_unavailable prevents a confident answer — but that sentence states the principle more clearly than anything currently written down here.

Where this stands otherwise: the storage/schema and fail-closed review you were owed is done and it passes — the three-DB-state migration (fresh, legacy-writable, legacy read-only) and the verdict ladder that never reaches current without indexed SHA, live HEAD, status availability and zero tracked/untracked. The only thing between this and merge is a clearance marker on the three cbm_popen calls in your test files, which is a maintainer step, not something you need to change.

Thanks for your patience — this has been open a long time, and it is the design that is being kept.

Clearance is done and I've now verified this end to end on today's main rather than on the August base: trial-merged onto fe85a6b2, built with ASan/UBSan, and ran git_context (10/10, seven of them yours), store_nodes 71/0, index_resilience 6/0, daemon_frontend 13/0, mcp 318/0, lint clean; then with the real binary in an isolated cache: index → current with the SHA matching HEAD, tracked edit → stale / tracked_changes_present, commit → stale / indexed_checkout_mismatch, untracked-only → unknown, re-index → current with the new SHA. The revert-check binds exactly where it should (never reading the persisted SHA reddens the store round-trip and four MCP verdict tests; dropping the tracked≠stale rule reddens the tracked tests and rename_counts_once). The fit is right too: one top-level freshness object, nothing else on the surface beyond the nullable indexed_checkout_sha column with its idempotent migration.

Two things are needed before it merges, both mechanical, and both caused by main moving under you:

1. One conflict, src/mcp/mcp.c:44-53 (the enum block): your branch adds MCP_STATUS_SAMPLE_MAX / MCP_REASONS_MAX, main added MCP_QUERY_MAX_VISIBLE_ROWS / MCP_OUTPUT_BYTES_PER_TOKEN_ESTIMATE in the same spot. Keep both.
2. Nine tests need "format":"json" in their index_status call: feat: make CLI and MCP output lean by default #1597 (merged this week) made the tree format the default for tool output, so JSON-substring assertions no longer match the default response. The calls are at tests/test_mcp.c:5711, 5736, 5843, 5881, 5937, 5988, 6046, 6076, 6131; main's own tool_index_status_includes_git_metadata shows the pattern. With that, mcp is 318/0. Line 5736 then runs past 100 columns, so run clang-format (the Homebrew LLVM one) over the file.

Non-blocking, take or leave: the verbose description at src/mcp/mcp.c:680-683 still says only "add worktree/shadow Git paths" — worth a clause about freshness; and recommended_action maps every unknown to reindex_to_record_indexed_checkout, which misleads for untracked_not_indexed / status_unavailable where the identity is recorded — a follow-up is fine.

Keep the sign-offs on the rebased commits and this merges on green. Thanks for the fast turnaround on the lint fix in August and for sticking with it.

Thanks for the CI report. I traced both failures to the same underlying issue: ci-ok was only aggregating the failed Unix shard, whose sole failing suite was complexity.

The failure was scheduler-sensitive, not a freshness regression: the shared-package scaling test observed nodes 244 -> 842 (ratio 3.45) while edges stayed 659 -> 1319 (ratio 2.00), exceeding the 2.75 node-ratio guard under the parallel wave. Re-running the suite locally produced the expected nodes 244 -> 484 (ratio 1.98) consistently.

Fix pushed in commit 6875483f: classify complexity as a serial/quiet-tail suite in scripts/run-tests-parallel.sh, alongside extraction, because both measure replicated-corpus scaling and are sensitive to concurrent allocator/CPU contention. No product logic or freshness behavior was changed.

Local verification on the updated source:

• git_context: 7 passed, 3 platform skips
• store_nodes: 71 passed
• index_resilience: 6 passed, 1 platform skip
• mcp: 315 passed, 11 platform skips
• complexity: 5 passed repeatedly

Please rerun the Unix shard/CI checks for 6875483f.

Thank you for tracking down the failing shard and recording both the parallel and isolated counts. That evidence is useful.

Please separate the serial-suite workaround in 6875483 from this freshness PR. The reported change from 484 to 842 nodes is a difference in graph contents, not just elapsed time; moving the suite to the quiet tail may avoid the symptom, but does not yet establish whether the cause is shared test state, extraction nondeterminism, or another defect. Please preserve the failing run and fixture details so we can investigate that discrepancy separately rather than treating a serial green run as a correctness fix.

This does not withdraw our agreement on the freshness design. We also owe you the final integration review after our earlier merge commitment. Thank you for the repeated updates and your patience.

Thank you for splitting the complexity-suite workaround out as we asked. The branch is now the freshness change alone, which is the version we want to merge.

main has moved on since, and the branch now conflicts in two files: src/pipeline/pipeline.c and src/pipeline/pipeline_internal.h. Most of the churn comes from today's markdown-link pass (#1832) and the memory-core work. Could you rebase onto the current main? Everything else merges cleanly.

One heads-up so it doesn't surprise you: a rebase produces a new head, so we re-run our security clearance and verification on it before merging. That is routine and nothing you need to do. The design agreement from 2026-09-01 (your freshness object is the one that ships) still stands. Thank you for your patience with how long this one has taken.

Thank you, @tmonestudio -- and thank you for splitting the complexity workaround out when we asked; the branch you left is exactly the freshness change we agreed to ship on 1 September. Since main had moved a long way, we finished the last mile here rather than ask you for another rebase. Your commit is untouched; we added four commits on top:

1. Merged current main (signed-off merge commit). The five conflicts were all places where main and your branch added code side by side -- main's new index_status "watch" object, the resource-policy accessors in the pipeline, an edges schema probe in the store, and new git-environment tests -- so both sides are kept. index_status now reports your "freshness" object next to "watch".
2. The worktree status probe now spawns git through cbm_popen_git instead of cbm_popen. Since pre-commit hook fails the test suite: git exports GIT_DIR into hooks, which overrides git -C in tests/test_pipeline.c #2003 every other git child does this: an inherited GIT_DIR/GIT_WORK_TREE (from a git hook or an editor) overrides "git -C", and on a real checkout that made a clean tree read as stale with another repository's changes. A new test, worktree_status_ignores_inherited_repo_env, pins it.
3. The status samples and the stored indexed_checkout_sha now allocate through the memory core (cbm_alloc / cbm_realloc / cbm_free / cbm_mem_strdup). Main added a ratchet that fails when a file gains raw malloc/free sites, and these were the only new ones.
4. clang-format over the new status parser (the project style keeps case labels unindented). Formatting only.

We re-ran it end to end on a real repository: clean -> current, tracked edit -> stale, new commit -> stale with the checkout mismatch, untracked-only -> unknown, re-index -> current with the new SHA. The fail-closed ladder you designed did all the work there. Thank you for sticking with this one for so long.

Thanks for your patience and for the freshness design. The October 20 update records that we took over the integration work, so there is no outstanding rebase request for you. Sorry for the delay on the remaining maintainer follow-up—I’ll follow up soon.

Thank you, @tmonestudio, for this change and for your patience while main kept moving under it. Your freshness object and the fail-closed verdict ladder are exactly what we agreed to ship, and your tests (the store round-trip, the rollback on a failed replace, and the git-backed freshness cases) carry over unchanged.

One thing first, because it changes something we said in public: on 5 September we wrote that the nullable indexed_checkout_sha column on index_coverage_meta, with its idempotent migration, was the right fit. It was when we wrote it. Since then main gained a graph digest for change-aware test selection that fingerprints index_coverage_meta column by column, and an extra column there makes that digest refuse every current database, so test selection falls back to running everything. Rather than change the digest, we moved the checkout SHA out of that table. Your design stays the same; only the place the value is stored changes.

We pushed these commits on top of your branch:

1. fix(store): keep the indexed checkout outside the digested tables. The checkout SHA now lives in a small table of its own, index_checkout (project, indexed_checkout_sha), written and cleared in the same transaction as the coverage metadata and removed with its project. Because it is a new table, existing databases need no migration, so the ALTER TABLE step is gone. The store API is unchanged: cbm_coverage_meta_t.indexed_checkout_sha reads and writes as before, and read-only opens of older databases still report no identity. A new test, store_graph_digest_indexed_checkout_identity_is_not_bound, writes a store with and without a checkout SHA and checks that it still digests and that the SHA leaves the digest unchanged.
2. Merged current main (signed-off merge commit). Five files conflicted, all places where main and your branch added code side by side: main's new unresolved_calls_complete field sits next to your indexed_checkout_sha in the coverage metadata struct, in its initializers in pipeline.c and pipeline_incremental.c, and in the store schema, and main's new coverage test sits next to yours in test_store_nodes.c. Both sides are kept everywhere. One spot needed more than that: main moved the coverage metadata write into a helper, coverage_write_metadata, so the two index_checkout writes moved into that helper with it. The rest of current main merged cleanly, including the index format bump for the new Next.js route support; your change needs no version bump of its own, because index_checkout is created whenever a database is opened for writing.

Please pull before pushing again, since the branch has new commits from us. Thank you again for the careful work and for sticking with this one for so long.

Merged - thank you very much, @tmonestudio! index_status now reports freshness from real checkout evidence: it compares the indexed checkout with the current HEAD and the tracked changes, and fails closed with a clear verdict and action instead of guessing. Your verdict ladder and the git-backed tests are exactly what we wanted, and thank you again for your patience while main moved under this branch for so long.